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IMAGE PROCESSING APPARATUS 

The present invention relates to the recording of images 
of an object and the computer processing of the image 
data to determine the positions and orientations at which 
the images were recorded and to generate a three- 
dimensional (3D) computer model of the object- More 
particularly, the present invention relates to the use of 
a patterned calibration object for imaging together with 
the object, and the computer processing of the image data 
to determine the imaging positions and orientations on 
the basis of the calibration object's pattern in the 
images and to generate data defining a 3D computer model 
of the object using the calculated positions and 
orientations . 

3D computer models of objects are useful for many- 
applications . In particular , 3D computer models are 
often used in computer games and for computer aided 
design (CAD) applications. In addition, there is now a 
growing demand to have 3D computer models of objects for 
uses such as the embellishment of Internet sites etc. 

Many methods are known for generating 3D computer models 
of objects. In particular, a method is known in which a 



subject object to be modelled is placed on a flat two- 
dimensional calibration object having a known pattern of 
features thereon, and images of the subject object 
together with the calibration object are recorded at 
different positions and orientations. Each recorded 
image is then processed to calculate the position and 
orientation at which is was recorded on the basis of the 
positions of the features in the calibration object's 
pattern in the image. Subsequently, a 3D computer model 
of the subject object is generated using the input images 
and the calculated positions and orientations. 

Examples of this method are described, for example, in 
"Automatic Reconstruction of 3D Objects Using A Mobile 
Camera" by Niem in Image and Vision Computing 17 (1999) 
pages 125-134, "The Lumigraph" by Gortler et al in 
Computer Graphics Proceedings, Annual Conference Series, 
1996 ACM-0-89791-764-4/96/008 , JP-A-9-170914 and the 
applicant 1 s earlier co-pending UK patent application 
0012 812.4 (the full contents of which are incorporated 
herein by cross-reference). 

The inventors in the present case have found that this 
known method suffers from a number of problems, however. 



More particularly, the inventions in the present case 
have found that 3D computer models generated using this 
method can be inaccurate. 

Accordingly , it is an object of the present invention to 
address this problem. 

The inventors of the present invention have recognised 
that inaccuracies in 3D computer models are caused by a 
number of factors as set out below. 

More particularly, shadows are often cast on the 
calibration object which appear to connect to the bottom 
of the subject object. Accordingly, in recorded images, 
the processing apparatus may not distinguish between the 
subject object and shadow, in which case the shadow is 
therefore determined to be part of the subject object. 
This problem is compounded because, if the subject object 
and calibration object are moved relative to the camera 
in order to record images at different positions and 
orientations, then the shadows move. On the other hand, 
if the camera itself is moved, the shadows cast by the 
camera move. 



Similarly, features in the pattern on the calibration 
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object often touch the subject object in a two- 
dimensional recorded image and are therefore determined 
by the processing apparatus to be part of the subject 
object . 

Similarly, features on the surface on which the subject 
object and calibration object are placed for imaging 
(such as marks on a floor or table) often touch the 
outline of the subject object in a recorded image, and 
are therefore determined by the processing apparatus to 
be part of the subject object. 

Further, it is often desirable to choose the colour of 
the calibration object to emphasise (contrast) the 
pattern of features thereon (to assist detection in input 
images by the processing apparatus). However, the colour 
of the calibration object is often reflected onto the 
subject object so that the 3D computer model is generated 
with an incorrectly coloured surface. 

Yet further, it is difficult to record images of the 
subject object at low elevation angles. This is because, 
for such angles, the pattern of features on the 
calibration object on which the subject object is sitting 
is distorted in the images to such an extent that 



detection of the features in input images by the 
processing apparatus in a reliable manner is not 
possible. By not being able to record images from low 
elevation angles, input images showing in detail parts of 
the subject object may not be available, resulting in 
those parts being inaccurately reproduced in the 3D 
computer model . 

Accordingly, it is an object of the present invention to 
address one or more of these problems - 

In terms of solutions, the inventors have recognised that 
the problem of shadows could be addressed by using 
diffuse illumination which is symmetrical about the 
subject object and calibration object. However, this" is 
very difficult and/or expensive to achieve in practice. 

Accordingly, it is an object of the present invention to 
provide a different solution. 

According to the present invention, there is provided a 
method of recording and processing images of a subject 
object to generate a three-dimensional computer model, in 
which the subject object is held in a fixed configuration 
relative to a calibration object by an object support 



while images are recorded at different camera positions 
and orientations, the image data is processed to 
determine the imaging positions and orientations on the 
basis of features from the calibration object visible in 
the images, and a three-dimensional computer model of the 
subject object alone or the subject object together with 
the object support is generated using the calculated 
positions and orientations. 

The present invention also provides apparatus components 
for carrying out the above method, including a processing 
apparatus and a computer program, and methods relating 
thereto . 



By imaging the subject object away from the surface of 
the calibration object (for example on an object stand) 
instead of placing the subject object directly on the 
calibration object, the user is provided with greater 
flexibility in the selection of the pattern on the 
calibration object, the selection of the colour of the 
calibration object and the selection of imaging positions 
and orientations. This is because the problem of 
features from the pattern on the calibration object 
appearing to touch the subject object in input images, 
the problem of the calibration object colour being 



reflected onto the subject object and the problem of 
recording input images form low elevation angles are all 
addressed by moving the subject object away from the 
calibration object . 

Preferably, the top surface of the object support on 
which the subject object is placed is shaped and sized so 
that there is no horizontal surface of any substantial 
size protruding from beneath the base of the subject 
object when the subject object is placed on the object 
support. In this way, the user is provided with greater 
flexibility in the selection of lighting conditions 
because the problem of shadows appearing as part of the 
subject object in input images is also addressed. 

Preferably, the subject object is imaged with a screen 
behind it. In this way, the user is provided with 
greater flexibility in the selection of the surface on 
which the calibration object, object support and subject 
object are placed for imaging because the problem of 
marks on the surface appearing as part of the subject 
object in input images is addressed. 



Embodiments of the invention will now be described, by 
way of example only, with reference to the accompanying 
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drawings, in which: 

Figure 1 schematically shows the components of an 
embodiment of the invention, together with the notional 
5 functional processing units into which the processing 

apparatus component can be thought of as being configured 
when programmed by programming instructions; 

Figure 2 illustrates the connection of a photographic mat 
10 to a mirror; 

Figure 3 illustrates the recording of images of a subject 
object for which a 3D computer model is to be generated, 
together with an object support and photographic mat; 
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Figures 4a-4e illustrate how the choice of camera viewing 
position and orientation and the choice of height of 
object support affect the input images recorded by the 
camera; 

Figure 5 illustrates images of the object, object support 
and photographic mat which are input to the processing 
apparatus in Figure 1 ; 



25 Figure 6 shows the processing operations performed by the 



processing apparatus in Figure 1 to process input data; 

Figure 7 shows the processing operations performed at 
step S6-8 in Figure 6; 

Figure 8 illustrates the definition of an initial volume 
at step S6-10 in Figure 6 on which to perform voxel 
carving operations to generate the 3D surface model of 
the subject object and object support; 

Figure 9 shows the processing operations performed at 
step S6-12 in Figure 6; 

Figure 10 illustrates the creation of pixel data for a 
virtual image at step S9-2 in Figure 9; 

Figure 11 shows the processing operations performed in 
the first embodiment at step S6-14 in Figure 6; 

Figures 12a, 12b and 12c illustrate the display and 
movement of the moveable plane at steps Sll-2 and Sll-4 
in Figure 11; 

Figure 13 illustrates the re-definition of an initial 
volume on which to perform voxel carving at step SI 1-8 in 



Figure 11; 



Figure 14 illustrates an object support having a 
transparent top portion in a second embodiment; 

Figure 15 shows the processing operations performed in 
the second embodiment at step S6-14 in Figure 6; 

Figure 16 shows an object support having a marker thereon 
to assist the user in selecting camera imaging positions 
and orientations in a further embodiment; 

Figure 17 illustrates an object support in a further 
embodiment, in which identifiable features are provided 
on the object support rather than on a photographic mat; 

Figure 18 illustrates a photographic mat and object 
support in a further embodiment, in which identifiable 
features are provided on both the photographic mat and 
object support; 

Figure 19 illustrates a further embodiment, in which the 
photographic mat is replaced by a three-dimensional 
calibration object having identifiable features thereon; 
and 



Figures 20a and 20b illustrate how a three-dimensional 
computer model may be edited using a plane of variable 
position and orientation in all embodiments. 

First Embodiment 

Referring to Figure 1, an embodiment of the invention 
comprises a processing apparatus 2, such as a personal 
computer, containing, in a conventional manner, one or 
more processors, memories, graphics cards etc, together 
with a display device 4, such as a conventional personal 
computer monitor, user input devices 6, such as a 
keyboard, mouse etc, a printer 8, and a display panel 10 
comprising a flat panel having controllable pixels, such 
as the PL400 manufactured by WACOM • 

The processing apparatus 2 is programmed to operate in 
accordance with programming instructions input, for 
example, as data stored on a data storage medium, such as 
disk 12, and/or as a signal 14 input to the processing 
apparatus 2, for example form a remote database, by 
transmission over a communication network (not shown) 
such as the Internet or by transmission through the 
atmosphere, and/or entered by a user via a user input 
device 6 such as a keyboard. 
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As will be described in more detail below, the 
programming instructions comprise instructions to cause 
the processing apparatus 2 to become configured to 
process input data defining a plurality of images of one 
or more subject objects recorded at different positions 
and orientations to calculate the positions and 
orientations at which the input images were recorded and 
to use the calculated positions and orientations to 
generate data defining a three-dimensional computer model 
of the subject object(s). In this embodiment, the 
subject object is placed on an object support and the 
subject object and support are imaged on a calibration 
object (a two-dimensional photographic mat in this 
embodiment) which has a known pattern of features 
thereon. The positions and orientations at which the 
input images were recorded are calculated by detecting 
the positions of the features of the calibration object 
pattern in the images. As will be explained in more 
detail below, by using an object support to image the 
subject object, accurate three-dimensional computer 
models of the subject object can be consistently 
obtained. 

When programmed by the programming instructions , 
processing apparatus 2 can be thought of as being 



configured as a number of functional units for performing 
processing operations. Examples of such functional units 
and their interconnections are shown in Figure 1. The 
units and interconnections illustrated in Figure 1 are, 
however, notional and are shown for illustration purposes 
only to assist understanding; they do not necessarily 
represent units and connections into which the processor, 
memory etc of the processing apparatus 2 become 
configured. 

Referring to the functional units shown in Figure 1, a 
central controller 20 processes inputs from the user 
input devices 6, and also provides control and processing 
for the other functional units. Memory 24 is provided 
for use by central controller 2 0 and the other functional 
units. 

Mat generator 30 generates control signals to control 
printer 8 or display panel 10 to print a photographic mat 
34 on a recording medium such as a piece of paper, or to 
display the photographic mat on display panel 10. As 
will be described in more detail below, the photographic 
mat comprises a predetermined pattern of features and the 
subject object (s) for which a three-dimensional computer 
model is to be generated is placed on an object support 



on the printed photographic mat 34 or on the display 
panel 10 on which the photographic mat is displayed. 
Images of the subject object(s), object support and the 
photographic mat are then recorded and input to the 
processing apparatus 2 . 

The colour of the recording medium on which printer 8 
prints the photographic mat is chosen to maximise the 
contrast between the recording medium and the features in 
the pattern of the photographic mat (thereby increasing 
the likelihood that the features will be located 
accurately in the input images by processing apparatus 
2). The colour can be freely chosen in this embodiment 
without having an adverse effect on the 3D computer model 
subsequently generated because the effect of light 
reflected from the printed photographic mat onto the 
subject object is minimised by placing the subject object 
on an object support and thereby raising the subject 
object away from the photographic mat. It has been found 
in practice that light , unsaturated colours give good 
results when chosen as the colour of the recording 
medium. 

Mat generator 30 stores data defining the pattern of 
features printed or displayed on the photographic mat for 



use by the processing apparatus 2 in calculating the 
positions and orientations at which the input images were 
recorded. More particularly, mat generator 30 stores 
data defining the pattern of features together with a 
coordinate system relative to the pattern of features 
(which, in effect, defines a reference position and 
orientation of the photographic mat), and processing 
apparatus 2 calculates the positions and orientations at 
which the input images were recorded in the defined 
coordinate system (and thus relative to the reference 
position and orientation) - 

In this embodiment, the pattern on the photographic mat 
comprises spatial clusters of features for example as 
described in co-pending UK patent application 0012812.4 
(the full contents of which are incorporated herein by 
cross-reference) or any known pattern of features, such 
as a pattern of coloured dots, with each dot having a 
different hue/brightness combination so that each 
respective dot is unique, for example as described in JP- 
A-9-170914, a pattern of concentric circles connected by 
radial line segments with known dimensions and position 
markers in each quadrant, for example as described in 
"Automatic Reconstruction of 3D Objects Using A Mobile 
Camera" by Niem in Image and Vision Computing 17 (1999) 
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pages 125-134, or a pattern comprising concentric rings 
with different diameters, for example as described in 
"The Lumigraph" by Gortler et al in Computer Graphics 
Proceedings, Annual Conference Series, 1996 ACM-0-89791- 
5 764-4/96/008 . 

In the remainder of the description of this embodiment, 
it will be assumed that the pattern is printed by printer 
8 on a recording medium (in this embodiment, a sheet of 
10 paper) to generate a printed photographic mat 34, 

although, as mentioned above, the pattern could be 
displayed on display panel 10 instead . 

Input data store 40 stores input data input to the 
15 processing apparatus 2 for example as data stored on a 

storage device, such as disk 42, as a signal 44 
transmitted to the processing apparatus 2, or using a 
user input device 6. The input data defines a plurality 
of images recorded at different positions and 
20 orientations showing one or more subject objects and an 

object support on the photographic mat. In this 
embodiment , the input data also defines an input image 
showing the background against which the subject 



object(s) and object support were imaged, together with 
2 5 part of the photographic mat to show the background 



colour thereof or a different object having the same 
colour as the background colour of the mat. In addition, 
in this embodiment, the input data also includes data 
defining the intrinsic parameters of the camera which 
recorded the images, that is, the aspect ratio, focal 
length, principal point (the point at which the optical 
axis intersects the imaging plane), first order radial 
distortion coefficient, and skew angle (the angle between 
the axes of the pixel grid; because the axes may not be 
exactly orthogonal) - 

The input data defining the input images may be generated 
for example by downloading pixel data from a digital 
camera which recorded the images, or by scanning 
photographs using a scanner (not shown). The input data 
defining the intrinsic camera parameters may be input by 
a user using a user input device 6. 

Camera calculator 50 processes each input image to detect 
the positions in the image of the features on the 
photographic mat and to calculate the position and 
orientation of the camera when the input image was 
recorded . 

Image data segment er 6 0 processes each input image to 
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separate image data corresponding to the subject object 
and object support from other image data in the image. 



Surface modeller 70 processes the segmented image data 
produced by image data segmenter 6 0 and the data defining 
the positions and orientations at which the images were 
recorded generated by camera calculator 50, to generate 
data defining a 3D computer model representing the actual 
surfaces of the subject object (s) and object support in 
the input images. 

Surface texturer 80 generates texture data from the input 
image data for rendering onto the surface model produced 
by surface modeller 70. 

Object/support separator 90 performs processing to amend 
the three-dimensional computer model of the subject 
object and object support generated by surface modeller 
7 0 to delete the part of the model corresponding to the 
object support, so as to leave a three-dimensional 
computer model of the subject object alone. 

Display processor 110, under the control of central 
controller 20, displays instructions to a user via 
display device 4. In addition, under the control of 
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central controller 20, display processor 110 also 
displays images of the 3D computer model form a user- 
selected viewpoint by processing the surface model data 
generated by surface modeller 70 and/or object/support 
5 separator 90 and rendering texture data produced by 

surface texturer 80 onto the surface model. 



Output data store 12 0 stores the camera positions and 
orientations calculated by camera calculator 50 for each 

10 input image, the image data relating to the subject 

object and object support from each input image generated 
by image data segmenter 60 , and also the surface model 
and texture data therefor generated by surface modeller 
70, object/support separator 90, and surface texturer 80. 

15 Central controller 20 controls the output of data from 

output data store 12 0, for example as data on a storage 
device, such as disk 122, and/or as a signal 124. 
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Referring to Figure 2, in this embodiment the pattern of 
features on the photographic mat 34 is printed by printer 
8 around a central blank area which is cut out to leave 
a hole 130 in the photographic mat 34. 

The photographic mat 34 with the hole 130 therein is then 
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be explained in more detail below, hole 130 and mirror 
140 enable the reflection of the bottom surface of the 
subject object to be seen in input images, and hence 
texture data for the bottom surface of the subject object 
in the three-dimensional computer model can be generated. 

Referring to Figure 3, the mirror 14 0 with the printed 
photographic mat 3 4 attached thereto is placed on a flat 
surface 200, and the subject object 210 for which a 3D 
computer model is to be generated is placed on an object 
support 220 on the photographic mat 34 so that the object 
support 220 is positioned in the hole 130 and is 
surrounded by the features making up the pattern on the 
mat . 

By raising subject object 210 off the photographic mat 34 
using object support 220, any shadows on the photographic 
mat 34 will touch the object support 220 and not the 
subject object 210. Similarly, features in the pattern 
on the photographic mat will appear to touch only the 
object support 220 and not the subject object 210 in 
recorded images. Accordingly in the 3D computer model 
generated by processing apparatus 2, errors may occur in 
the model of the ~ object ~ supporT "220 but the subject 
object 210 will be accurately modelled. However, the 



errors in the object support 220 are unimportant and, in 
this embodiment, object/support separator 90 removes the 
3D computer model of the object support 220 leaving a 3D 
computer model of the subject object 210 on its own. 

Yet further, as will be explained below, by raising the 
subject object 210 above the photographic mat 34, images 
at low elevation angles can be recorded in which the 
pattern of features on the photographic mat 34 is not 
distorted to such an extent that processing apparatus 2 
cannot perform processing to identify the features in the 
input images and calculate the imaging positions and 
orientations. Accordingly, the accuracy of the 3D 
computer model of the subject object 210 is further 
increased because images showing the necessary detail of 
the subject object 210 can be recorded and processed. 

In this embodiment, the object support 220 comprises a 
horizontal plate 222 of a transparent material, such as 
glass, supported by four cylindrical opaque legs 224. 
Accordingly, because the horizontal plate 222 is 
transparent, the bottom surface of the subject object 210 
is visible as a reflection in the mirror 140. 

A background screen 228 of a substantially uniform colour 



is placed behind the subject object 210 and object 
support 220. In this way, no part of the subject object 
210 (or object support 220) appears against the surface 
200 in recorded images. Instead, the subject object 210 
appears only against the background screen. This allows 
the user to choose freely the surface 200 on which the 
photographic mat 34 is placed, because marks on the 
surface 200 or regions of non-uniform colour on the 
surface 200 , etc . will not touch the outline of the 
subject object 210 in any input image. Accordingly, as 
will be explained in more detail below, the use of 
background screen 228 assists in separating image data 
relating to the subject object 210 and object support 220 
from other image data during segmentation processing by 
image data segmenter 60. 

Images of the subject object 210, object support 220 and 
photographic mat 34 are recorded at different positions 
and orientations to show different parts of subject 
object 210 using a digital camera 230. In this 
embodiment, data defining the images recorded by camera 
230 is input to processing apparatus 2 as a signal 44 
along wire 232. 

More particularly, in this embodiment, camera 230 and 
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background screen 228 remain in fixed positions and 
photographic mat 34 with the subject object 210 and 
object support 220 thereon is moved (translated) and 
rotated (for example in the direction of arrow 240) on 
surface 200, and photographs of the subject object 210 at 
different positions and orientations relative to the 
camera 230 are recorded. During the rotation and 
translation of the photographic mat 34 on surface 200, 
the subject object 210 and object support 2 20 do not move 
relative to the mat 34. 

Figures 4a to 4e illustrate how, in this embodiment, the 
location and viewing angle of camera 2 30 are chosen and 
how the height of the object support 220 is chosen. 

Referring to Figure 4a, if camera 230 is located in 
position (i), that is, a highly elevated position looking 
down on the subject object 210, then, as shown in Figure 
4b, in each image recorded by camera 230, the top part of 
the subject object 210 will be outlined against the 
background screen 228, but the bottom part of the subject 
object 210 will be outlined against the photographic mat 
34 and/or the horizontal plate 222 of the object support 
220. This is undesirable because any shadows on the 
horizontal plate 222 and photographic mat 34 may appear 



24 2714301 
as part of the subject object 210 and hence may not be 
separated from the subject object 210 by image data 
segmenter 60. Similarly, one or more features in the 
pattern on the photographic mat 34 may touch the subject 
object 210 in an input image (and hence appear to be part 
of the subject object 210), and may not therefore be 
separated from the subject object 210 by image data 
segmenter 60 . 

Accordingly, a more desirable imaging location and 
viewing direction is that marked (ii) in Figure 4a, in 
which the optical axis of the camera 230 is substantially 
in line with the horizontal plate 222 of the object stand 
220. However, if the height of the horizontal plate 222 
above photographic mat 34 is too low (as is the case in 
the example shown in Figure 4a), then, as shown in Figure 
4c, in each input image recorded by camera 230, the 
photographic mat 3 4 will appear distorted to such an 
extent that the features in the pattern thereon can not 
be distinguished from each other. 

Consequently, as shown in Figure 4d, the position and 
viewing direction of camera 230 are chosen so that the 
optical axis of the camera is substantially in line with 
the horizontal plate 222 of the object support 220, and 
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the height of the object support 220 is chosen so that 
the pattern of features on the photographic mat 34 is not 
distorted in the input images. In this way, referring to 
Figure 4e, which shows an example of an input image 
recorded using the configuration shown in Figure 4d, the 
subject object 210 is surrounded by the background screen 
228 (thereby enabling image data segmenter 6 0 to segment 
image data relating to the subject object 210 more 
accurately) and the features in the pattern on the 
photographic mat 3 4 can be distinguished from each other 
(thereby enabling camera calculator 50 to detect the 
position of each feature in an input image and hence 
determine the position and orientation at which the input 
image was recorded) . It should be noted that the 
background screen 228 does not need to extended to the 
edges of the input images (as is the case in the example 
shown in Figure 4e), but only needs to be extended above, 
below and to the sides of the subject object 210 so that 
the subject object 210 is surrounded thereby. 

In summary, by raising the subject object 210 off the 
photographic mat 34 (this being achieved in this 
embodiment using object support 220) input images can be 
recorded in which shadows on the photographic mat 3 4 and 
features in the pattern on photographic mat 34 do not 



touch the outline of the subject object 210. Further, 
the colour of the photographic mat 34 can be freely 
chosen to emphasise the features in the pattern thereon 
because the colour is reflected much less onto the 
subject object 210. Further, images of the subject 
object 210 can be recorded at low elevation angles, 
ensuring that suitable images are available to model all 
parts of the subject object 210 accurately in the 3D 
computer model. 

In addition, by placing background screen 22 8 behind the 
subject object 210, the choice of surface 200 can be 
freely made because input images can be recorded in which 
no part of the surface 200 is seen to touch the subject 
object 210. 

Figure 5 shows examples of input images 300, 302, 304 and 
306 of the subject object 210, object support 220 and 
photographic mat 34 in different positions and 
orientations relative to camera 230. 

In this embodiment, following the recording and input of 
images of subject object 210, object support 220 and 
photographic mat 34 , a further image is recorded and 
input to processing apparatus 2 . This further image 



comprises a "background image", which is an image of the 
surface 200, background screen 228 and an object having 
the same colour as the paper on which photographic mat 34 
is printed. Such a background image may be recorded by 
placing a blank sheet of paper having the same colour as 
the sheet on which photographic mat 34 is recorded on 
surface 200 in front of the background screen 228, or by 
turning the photographic mat 34 over on surface 200 so 
that the pattern thereon is not visible in the image. 

Figure 6 shows the processing operations performed by 
processing apparatus 2 to process input data in this 
embodiment . 

Referring to Figure 6, at step S6-2, central controller 
20 causes display processor 110 to display a message on 
display device 4 requesting the user to input data for 
processing . 

At step S6-4, data input by the user in response to the 
request at step S6-2 is stored in the input data store 
40. More particularly, in this embodiment, the input 
data comprises image data defining the images of the 
subject object 210, object support 220 and mat 34 
recorded at different positions and orientations relative 
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to the camera 230, the "background image" showing the 
surface 200 on which photographic mat 34 was placed to 
record the input images together with the background 
screen 228 and an object having the same colour as the 
recording material on which the pattern of photographic 
mat 34 is printed, and data defining the intrinsic 
parameters of the camera 230 which recorded the input 
images, that is the aspect ratio, focal length, principal 
point (the point at which the optical axis intersects the 
imaging plane), the first order radial distortion 
coefficient, and the skew angle (the angle between the 
axes of the pixel grid) . 

At step S6-6, camera calculator 50 processes the input 
data stored at step S6-4 to determine the position and 
orientation of the camera 230 relative to the 
photographic mat 34 (and hence relative to the subject 
object 210 and object support 22 0) for each input image. 
This processing comprises, for each input image, 
detecting the features in the image which make up the 
pattern on the photographic mat 34 and comparing the 
features to the stored pattern for the photographic mat 
to determine the position and orientation of the camera 
230 relative to the mat. The processing performed by 
camera calculator 50 at step S6-6 depends upon the 



pattern of features used on the photographic mat 34. 
Accordingly, suitable processing is described, for 
example, in co-pending UK patent application 0012812.4, 
JP-A-9-170914 , "Automatic Reconstruction of 3D Objects 
Using A Mobile Camera" by Niem in Image and Vision 
Computing 17 (1999) pages 125-134 and "The Lumigraph" by 
Gortler et al in Computer Graphics Proceedings , Annual 
Conference Series, 1996 ACM-0-8979 1-764-4/96/008 . 

At step S6-8, image data segmenter 60 processes each 
input image to segment image data representing the 
subject object 210 and object support 220 from image data 
representing the photographic mat 34, the surface 200 on 
which the mat 34 is placed and the bakcground screen 228 
(step S6-8 being a preliminary step in this embodiment to 
generate data for use in the subsequent generation of a 
3D computer model of the surface of subject object 210 
and object support 220, as will be described in more 
detail below) . 

Figure 7 shows the processing operations performed by 
image data segmenter 60 at step S6-8. 

Referring to Figure 7, at steps S7-2 to S7-10, image data 
segmenter 60 builds a hash table of quantised values 



representing the colours in the input images which 
represent the photographic mat 34, the surface 200 and 
the background screen 228 but not the subject object 210 
and object support 220. 

More particularly, at step S7-2, image data segmenter 60 
reads the RBG data values for the next pixel in the 
"background image" stored at step S6-4 in Figure 6 (that 
is, the final image to be input to processing apparatus 
2 which shows the surface 200, the background screen 228 
and an object having the same colour as the material on 
which photographic mat 34 is printed). 

At step S7-4, image data segmenter 60 calculates a 
quantised red (R) value, a quantised green (G) and a 
quantised blue (B) value for the pixel in accordance with 
the following equation: 

(p+t/2) 

4- t t 1 ) 

where: "q" is the quantised value; 

"p M is the R, G or B value read at step S7-2; 

"t" is a threshold value determining how near 
RGB values from an input image showing the 
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subject object 210 need to be to background 
colours to be labelled as background. In this 
embodiment, "t" is set to 4. 

At step S7-6, image data segmenter 60 combines the 
quantised R, G and B values calculated at step S7-4 into 
a "triple value" in a conventional manner . 

At step S7-8, image data segmenter 60 applies a hashing 
function to the quantised R, G and B values calculated at 
step S7-4 to define a bin in a hash table, and adds the 
"triple" value defined at step S7-6 to the defined bin. 
More particularly, in this embodiment, image data 
segmenter 60 applies the following hashing function to 
the quantised R, G and B values to define the bin in the 
hash table: 

h (q)=(qr ed & 7)*2~ S+ (g green & 7)*2~ 3+ (q blue & 7) ( 2 ) 

That is, the bin in the. hash table is defined by the 
three least significant bits of each colour. This 
function is chosen to try and spread out the data into 
the available bins in the hash table, so that each bin 
has only a small number of "triple" values. In this 
embodiment, at step S7-8, the "triple" value is added to 
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the bin only if it does not already exist therein, so 
that each "triple" value is added only once to the hash 
table . 

At step S7-10, image data segmenter 60 determines whether 
there is another pixel in the background image. Steps 
S7-2 to S7-10 are repeated until each pixel in the 
"background" image has been processed in the manner 
described above. As a result of this processing, a hash 
table is generated containing values representing the 
colours in the "background" image. 

At steps S7-12 to S7-48, image data segmenter 60 
considers each input image in turn and uses the hash 
15 table to segment the data in the input image relating to 

the photographic mat 34, the background screen 228 and 
the surface 200 from the data in the input image relating 
to the subject object 210 and object support 220. 

2 0 In this embodiment, the "background" image processed at 

steps S7-2 to S7-10 to generate the hash table does not 
show the features on the photographic mat 34. 
Accordingly, the segmentation performed at steps S7-12 to 
S7-46 does not distinguish pixel data relating to the 

25 subject object 210 and object support 220 from pixel data 



5 



10 
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relating to a feature on the mat 34, Instead, in this 
embodiment, the processing performed by surface modeller 
70 to generate the 3D computer model of the surface of 
subject object 210 and object support 220 is carried out 
in such a way that pixels relating to a feature on 
photographic mat 34 do not contribute to the surface 
model, as will be described in more detail below. 

At step S7-12, image data segmenter 60 considers the next 
input image, and at step S7-14 reads the R, G and B 
values for the next pixel in the input image (this being 
the first pixel the first time step S7-14 is performed) . 

At step S7-16, image data segmenter 60 calculates a 
quantised R value, a quantised G value and a quantised B 
value for the pixel using equation (1) above. 

At step S7-18, image data segmenter 60 combines the 
quantised R, G and B values calculated at step S7-16 into 
a "triple value". 

At step S7-20, image data segmenter 60 applies a hashing 
function in accordance with equation (2) above to the 
quantised values calculated at step S7-16 to define a bin 
in the hash table generated at steps S7-2 to S7-10. 
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At step S7-22, image data segmenter 60 reads the "triple' 1 
values in the hash table bin defined at step S7-20, these 
"triple" values representing the colours of the material 
of the photographic mat 34, the background screen 228 and 
5 the surface 200. 

At step S7-24, image data segmenter 60 determines whether 
the "triple" value generated at step S7-18 of the pixel 
in the input image currently being considered is the same 
10 as any of the background "triple" values in the hash 

table bin. 

If it is determined at step S7-24 that the "triple" value 
of the pixel is the same as a background "triple" value, 
15 then, at step S7-26, it is determined that the pixel is 

a background pixel and the value of the pixel is set to 
"black" . 

On the other hand, if it is determined at step S7-2 4 that 
2 0 the "triple" value of the pixel is not the same as any 

"triple" value of the background, then, at step S7-28, it 
is determined that the pixel is part of the subject 
object 210 or object support 220 and image data segmenter 
60 sets the value of the pixel to "white". 

25 
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At step S7-30, image data segmenter 60 determines whether 
there is another pixel in the input image. Steps S7-14 
to S7-30 are repeated until each pixel in the input image 
has been processed in the manner described above. 

At steps S7-32 to S7-44, image data segmenter 60 performs 
processing to correct any errors in the classification of 
image pixels as background pixels or object pixels. 

More particularly, at step S7-32, image data segmenter 60 
defines a circular mask for use as a median filter. In 
this embodiment, the circular mask has a radius of 4 
pixels . 

At step S7-34, image data segmenter 60 performs 
processing to place the centre of the mask defined at 
step S7-32 at the centre of the next pixel in the binary 
image generated at steps S7-26 and S7-28 (this being the 
first pixel the first time step S7-34 is performed) . 

At step S7-36, image data segmenter 60 counts the number 
of black pixels and the number of white pixels within the 
mask. 



At step S7-38, image data segmenter 60 determines whether 



the number of white pixels within the mask is greater 
than or equal to the number of black pixels within the 
mask . 

If it is determined at step S7-38 that the number of 
white pixels is greater than or equal to the number of 
black pixels, then, at step S7-40 image data segmenter 60 
sets the value of the pixel on which the mask is centred 
to white. On the other hand, if it is determined at step 
S7-38 that the number of black pixels is greater than the 
number of white pixels then, at step S7-42, image data 
segmenter 60 sets the value of the pixel on which the 
mask is centred to black. 

At step S7-44, image data segmenter 60 determines whether 
there is another pixel in the binary image, and steps S7- 
34 to S7-44 are repeated until each pixel has been 
processed in the manner described above. 

At step S7-48, image data segmenter 60 determines whether 
there is another input image to be processed. Steps S7- 
12 to S7-48 are repeated until each input image has been 
processed in the manner described above. 

Referring again to Figure 6, at step S6-10, surface 
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modeller 70 performs processing to generate data defining 
a 3D computer model of the surface of subject object 210 
and the surface of object support 220. 

In this embodiment, the processing at step S6-10 is 
performed in a conventional manner, and comprises the 
following three stages: 

(1) The camera positions and orientations generated at 
step S6-6 and the segmented image data generated at 
step S6-8 is processed to generate a voxel carving, 
which comprises data defining the 3D grid of voxels 
enclosing the surface of subject object 210 and the 
surface of object support 220. Surface modeller 70 
performs processing for this stage in a 
conventional manner, for example as described in 
"Rapid Octree Construction from Image Sequences" by 
R. Szeliski in CVGIP: Image Understanding, Volume 
58, Number 1, July 1993, pages 23-32. However, 
referring to Figure 8, in this embodiment, the 
start volume defined by surface modeller 70 on 
which to perform the voxel carve processing 
comprises a cuboid 400 having vertical side faces 
402, 404, 406, 408, a horizontal top face 410 and a 
horizontal bottom face 412. The vertical side 
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faces 402, 404, 406, 408 are positioned so that 
they touch the edge of the pattern of features on 
the photographic mat 34 (as schematically 
represented by the outer dotted circle in Figure 
8). Accordingly, the vertical side faces wholly 
contain the subject object 210. The position of 
the top face 410 is defined by intersecting a line 
42 0 from the focal point of the camera 230 through 
the top edge of any one of the input images stored 
at step S6-4 with a vertical line 422 through the 
centre of the photographic mat 34. More 
particularly, the focal point of the camera 230 and 
the top edge of an image are known as a result of 
the position and orientation calculations performed 
at step S6-6 and, by setting the height of the top 
face 410 to correspond to the point where the line 
420 intersects a vertical line 422 through the 
centre of the photographic mat 34, the top face 410 
will always be above the top of the subject object 
210 (provided that the top of the subject object 
210 is visible in each input image). The position 
of the horizontal base face 412 is set to be 
slightly above the plane of the photographic mat 
34. By setting the position of the base face 412 
in this way, features in the pattern on the 



photographic mat 3 4 (which were not separated from 
the subject object 210 and object support 220 in 
the image segmentation processing performed at step 
S6-8) will be disregarded during the voxel carving 
processing and a 3D surface model of the subject 
object 210 and object support 220 will be 
generated. 

The data defining the voxel carving is processed to 
generate data defining a 3D surface mesh of 
triangles defining the surface of the subject 
object 210 and the surface of the object support 
220. In this embodiment, this stage of the 
processing is performed by surface modeller 70 in 
accordance with a conventional marching cubes 
algorithm, for example as described in W.E.Lorensen 
and H.E.Cline: "Marching Cubes: A High Resolution 
3D Surface Construction Algorithm", in Computer 
Graphics, SIGGRAPH 87 proceedings, 21: 163-169, 
July 1987, or J. Bloomenthal: "An Implicit Surface 
Polygonizer " , Graphics Gems IV, AP Professional, 
1994, ISBN 0123361559, pp 324-350. 



The number of triangles in the surface mesh 
generated at stage 2 is substantially reduced by 
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performing a decimation process. 

In stage 3, surface modeller 70 performs processing in 
this embodiment to carry out the decimation process by 
randomly removing vertices from the triangular mesh 
generated in stage 2 to see whether or not each vertex 
contributes to the shape of the surface of subject object 
210 or the surface of object support 220. Vertices which 
do not contribute to the shape are discarded from the 
triangulation, resulting in fewer vertices (and hence 
fewer triangles) in the final model. The selection of 
vertices to remove and test is carried out in a random 
order in order to avoid the effect of gradually eroding 
a large part of the surface by consecutively removing 
neighbouring vertices. The decimation algorithm 

performed by surface modeller 70 in this embodiment is 
described below in pseudo-code. 

INPUT 

Read in vertices 

Read in triples of vertex IDs making up triangles 

PROCESSING 

Repeat NVERTEX times 

Choose a random vertex V, which hasn't been chosen 
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before 

Locate set of all triangles having V as a vertex, S 
Order S so adjacent triangles are next to each other 
Re-triangulate triangle set, ignoring V (i.e. remove 
5 selected triangles & V and then fill in hole) 

Find the maximum distance between V and the plane of 

each triangle 

If (distance < threshold) 

Discard V and keep new triangulation 

10 Else 

Keep V and return to old triangulation 



OUPUT 

Output list of kept vertices 
15 Output updated list of triangles 



Since the absolute positions of the features on 
photographic mat 34 are known (the features having been 
printed in accordance with prestored data defining the 
2 0 positions), the 3D computer model of the surface of 

subject object 210 and the surface of object support 220 
is generated at step S6-10 to the correct scale. 



25 



At step S6-12, surface texturer 80 processes the input 
image data stored at step S6-4 to generate texture data 
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for each triangle in the surface model generated by 
surface modeller 70 at step S6-10. 

Figure 9 shows the processing operations performed by 
5 surface texturer 90 at step S6-12 in this embodiment. 

Referring to Figure 9, at step S9-2, surface texturer 90 
generates image data which can be used as texture data 
for the bottom surface of the subject object 210, that is 

10 the surface which sits on the horizontal plate 222 of the 

object support 220. In this embodiment, this processing 
is performed by converting each input image of the 
subject object 210 stored at step S6-4 into a virtual 
image having a camera viewing position and orientation 

15 such that the underside of the subject object 210 can be 

seen and comprising the pixel data from the input image 
which corresponds to the reflection of the underside of 
the subject object 210 in the mirror 140. More 
particularly , as noted previously, mat generator 3 0 

2 0 stores data defining a coordinate system and camera 

calculator 50 calculates the position and orientation at 
which each input image was recorded in the defined 
coordinate system. In this embodiment, the centre of the 
photographic mat 34 is the origin of the coordinate 

25 system and the z-direction is the direction perpendicular 



to the plane of the photographic mat, pointing up. At 
step S9-2, surface texturer 80 processes each input image 
by changing the position and orientation of the input 
image calculated by camera calculator 50 to define a new 
position and orientation for the virtual image 
corresponding to a position and orientation which shows 
the underside of a subject object 210. More 
particularly, representing the position and orientation 
of the input image calculated by camera calculator 50 as 
a 3 by 4 matrix M, surface texturer 80 generates a 3 by 
4 matrix M' defining the position and orientation of the 
virtual image as follows: 



M' 



M 



10 0 0 
0 10 0 
0 0-10 
0 0 0 1 



(3) 



Surface texturer 80 generates pixel data for each virtual 
image by masking out all pixels in the corresponding 
input image which do not correspond to the hole 130 in 
the photographic mat 34 (the position of hole 130 in the 
input image being known from the position and orientation 
calculations performed by camera calculator 50 at step 
S6-6). In this way, referring to Figure 10, the pixel 
data for a virtual image comprises pixel data showing the 



reflection in the mirror 140 with all of the remaining 
pixel data masked out (indicated by the shading in Figure 
10) . 

It will be appreciated that, in some input images, the 
legs 224 of the object support 220 will appear in the 
reflection in mirror 140 and hence will appear in the 
corresponding virtual image. However, this is not a 
problem in this embodiment since, as will be clear from 
the description below, the texture data for the different 
triangles in the 3D computer model making up the bottom 
surface of subject object 210 does not necessarily come 
from the same input image, and therefore the effect of 
the legs 224 being visible in some of the input images 
will not be significantly noticeable. Alternatively, the 
legs 24 may be made of a transparent material, so that 
they are not visible in the reflection from mirror 140, 

Referring again to Figure 9, at steps S9-4 to S9-10, 
surface texturer 80 performs processing in a conventional 
manner to select each triangle in the surface mesh 
generated at step S6-10 and to define texture data from 
one of the input images for a selected triangle. 



More particularly, at step S9-4, surface texturer 80 
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considers the next triangle in the surface mesh generated 
at step S6-10 (this being the first triangle the first 
time step S9-4 is performed) and, at step S9-6, performs 
processing to find the input image "i" which is most 
front-facing to the triangle. 

More particularly, at step S9-6, surface texturer 80 
finds the input image for which the value nt.vi is 

largest, where fit is the triangle normal and Vi is the 

viewing direction for the "i"th image- This identifies 
the input image in which the surface triangle has the 
largest projected area. 

At step S9-8, surface texturer 80 .projects the triangle 
into the input image identified at step S9-6, and stores 
the vertices of the projected triangle as texture 
coordinates defining an image texture map in a 
conventional manner . 

At step S9-10, surface texturer 80 determines whether 
there is another triangle in the surface mesh, and steps 
S9-4 to S9-10 are repeated until each triangle has been 
processed in the manner described above. 

The result of performing the processing described above 



is a VRML (or similar format) model of the surface of 
subject object 210 and the object support 220, complete 
with texture coordinates defining image data to be 
rendered onto the model. 

Referring again to Figure 6, at step S6-14, 
object/support separator 90 processes the data defining 
the 3D computer model of the surface of subject object 
210 and the surface of object support 220 generated by 
surface modeller 70 at step S6-10 to remove the object 
support 220 from the model, leaving a 3D computer model 
of the subject object 210 alone. 

In this embodiment, object/support separator 90 performs 
processing in accordance with user instructions input via 
a user input device 6 to remove interactively the object 
support 220 from the 3D computer model. 

Figure 11 shows the processing operations performed by 
object/support separator 90 at step S6-14. 

Referring to Figure 11, at step Sll-2, object/support 
separator 90 causes display processor 110 to display on 
display device 4 an image of the 3D computer model of the 
subject object 210 and object support 220 generated by 
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surface modeller 70 rendered with texture data generated 
by surface texturer 80. In addition, referring to Figure 
12a, object/support separator 90 causes display processor 
110 to display a horizontal plane 450 intersecting the 
displayed 3D computer model, the horizontal plane 450 
being movable by the user in a vertical direction (that 
is, in a direction perpendicular to the photographic mat 
34) using a user input device 6 such as a mouse. 

At step Sll-4, referring to Figures 12a, 12b and 12c, 
object/support separator 90 moves the plane 450 up and/or 
down in a vertical direction in accordance with user 
input instructions to a final position defined by the 
user where the plane 450 separates the subject object 210 
and object support 22 0 in the 3D computer model (Figure 
12c). In addition, object/support separator 90 causes 
display processor 110 to display the 3D computer model of 
the subject object 210, object support 220 and plane 450 
from one or more different viewing positions and 
directions in accordance with user input instructions. 
This enables the user to view the 3D computer model from 
different viewpoints and move the plane 450 while viewing 
the 3D computer model from these viewpoints in order to 
ensure that the final position of the plane 450 
accurately separates the subject object 210 from the 
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object support 220 in the 3D computer model (plane 450 
always being constrained to move in a direction 
perpendicular to the calculated plane of the photographic 
mat 34, which may not be a vertical direction on the 
display device 4 when the 3D computer model is viewed 
from different viewing positions and directions). 

At step Sll-6, in response to a user input signal 
indicating that the user has finished moving the plane 
450, so that the plane is now in a position separating 
the subject object 210 and object support 220, 
object/support separator 90 stores the coordinates 
defining the final position of the plane. 

At step Sll-8, object/support separator 90 performs 
processing to delete the part of the 3D surface model 
generated by surface modeller 7 0 lying below the final 
position of the plane stored at step Sll-6 (that is, to 
delete the part of the model lying on the side of the 
plane 450 towards the calculated plane of the 
photographic mat 34). 

Referring to Figure 13, to perform the processing at step 
Sll-8, object/support separator 90 repeats the processing 
performed by surface modeller 70 at step S6-10, but this 
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time amending the cuboid 400 defining the start volume 
for the voxel carve processing so that the horizontal 
bottom face 412 is moved to a position corresponding to 
the final position of the plane 450 stored at step Sll-6. 
In this way, when the voxel carve processing and 
subsequent processing (described previously) is 
performed, a 3D computer model of the surface of the 
subject object 210 alone is generated, and the model does 
not include any part of the object support 220. 

Referring again to Figure 11, at step Sll-10, 
object/support separator 90 causes display processor 110 
to display an image of the 3D surface model generated at 
step SI 1-8 rendered with the appropriate texture data 
generated by surface texturer 80 at step S6-12 on display 
device 4 . 

At step Sll-12, object /support separator 90 determines 
whether a user input signal has been received indicating 
that further changes are to be made to the 3D computer 
model to remove the model of the object support 220. 
Steps Sll-4 to Sll-12 are repeated until it is determined 
at step Sll-12 that no further changes are necessary. 



Referring again to Figure 6, at step S6-16, central 
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controller 20 outputs the data defining the 3D computer 
model of the subject object 210 and texture data therefor 
from output data store 120, for example as data stored on 
a storage device such as disk 122 or as a signal 124 
(Figure 1). In addition, or instead, central controller 
20 may cause display processor 110 to display an image of 
the 3D computer model of the subject object 210 rendered 
with the texture data in accordance with a viewpoint 
input by a user, for example using a user input device 6. 
Alternatively, the data defining the position and 
orientation of the camera 230 for each input image 
generated at step S6-6 and the data defining the 
segmentation of each input image generated at step S6-8 
may also be output, for example as data recorded on a 
storage device such as disk 122 or as a signal 124. This 
data may then be input into a separate processing 
apparatus programmed to perform steps S6-10 to S6-14. 

In the first embodiment described above, object/support 
separator 9 0 processes the 3D computer model of the 
subject object and object support at step S6-14 to remove 
the object support therefrom in accordance with user 
input instructions defining the position of a plane 450 
to separate the subject object and the object support. 



However, a 3D computer model of the subject object alone 
(that is, without the object support) may be generated in 
different ways by object/support separator 90, as will be 
clear from the further embodiments described below. 

Second Embodiment 

A second embodiment of the invention will now be 
described. The components of the second embodiment and 
the processing operations performed thereby are the same 
as those in the first embodiment, with the exception that 
the object support is different and the processing 
operations performed by the object/support separator 90 
are different, as will be described below. In addition, 
in the second embodiment texture data for the underside 
of subject object 210 is not generated from the input 
images by surface texturer 8 0 (this being inessential 
because the underside of the subject object 210 is of no 
significant interest to the user in many cases). 

Referring to Figure 14, in the second embodiment, an 
object support 460 comprises a cylinder having an opaque 
bottom portion 470 and a transparent top portion 480. 

The diameter of the object support 460 is sized so that 



the subject object 210 covers the entire top surface of 
the object support 460 when it is placed thereon. In 
this way, the horizontal surface on which the subject 
object 210 sits does not protrude beyond the edges of the 
base of the subject object 210. Accordingly, there is no 
horizontal surface adjacent the subject object 210 on 
which shadows can be formed, and therefore no shadows can 
touch the subject object 210 in an input image. 

Because the top portion 480 of the object support 460 is 
transparent, in each input image stored at step S6-4, 
part of the background screen 228 will be visible between 
subject object 210 and the opaque bottom portion 470 of 
the object support 4 60, as shown in Figure 14. In this 
way, in each input image, the subject object 210 will 
appear to be separated from the bottom portion 470 of the 
object support 460. 

Consequently, in the processing performed by image data 
segmenter 60 at step S6-8, pixels relating to the portion 
of the background screen 228 between the subject object 
210 and the bottom portion 470 of the object support 460 
are designated as "background" pixels, while the pixels 
relating to the subject object 210 and the pixels 
relating to the bottom portion 470 of the object support 
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460 are designated as separated regions of "object" 
pixels. 

Therefore, in the processing performed by surface 
modeller 70 at step S6-10, a 3D computer model of the 
subject object 210 is generated and a separate, 
unconnected 3D computer model of the bottom portion 470 
of the. object support 460 is generated. 

Accordingly, in the processing performed by 
object/support separator 90 at step S6-14, the 3D 
computer model of the bottom portion 470 of the object 
support 460 merely needs to be distinguished from the 3D 
computer model of the subject object 210 and deleted. 

Figure 15 shows the processing operations performed by 
object/support separator 90 at step S6-14 in the second 
embodiment . 

Referring to Figure 15, at step S15-2, object/support 
separator 90 causes display processor 110 to display on 
display device 4 an image of the 3D computer model of the 
subject object 210 and the 3D computer model of the 
bottom portion 470 of the object support 460 generated by 
surface modeller 70 rendered with texture data generated 



by surface texturer 80. 



At step S15-4, object/support separator 90 causes display 
processor 110 to display a message on display device 4 
requesting the user to identify the 3D model of the 
bottom portion 470 of the object support 460. 

At step S15-6, in response to a signal from the user 
input via a user input device 6 such as a mouse, 
object/support separator 90 deletes the 3D computer model 
defined in the input signal, thereby leaving the 3D 
computer model of the subject object 210 alone. 

As a modification to this embodiment, the top portion 480 
of the object support 46 0 may be opaque and have the same 
colour as the background screen 228. This has the same 
effect as a transparent top portion 480 because, in each 
input image, the top portion 480 is indistinguishable 
from the background screen 228. 

As an alternative to the processing described above with 
reference to Figure 15 , object /support separator 9 0 may 
delete the 3D computer model of the bottom portion 470 of 
the object support 4 60 automatically ( that is , without 
input from the user) . More particularly, this is 
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achieved by deleting the 3D computer model which is 
closest to the position of the photographic mat 34 
calculated by camera calculator 50 at step S6-6, 
resulting in the correct 3D computer model being deleted 
5 every time because the subject object 210 is always 

placed on the object support 460 and is therefore always 
further away from the photographic mat 34 than the bottom 
portion 470 of the object support 460. 

10 Third Embodiment 

A third embodiment of the invention will now be 
described. The components of the third embodiment and 
the processing operations performed thereby are the same 

15 as those in the first embodiment, with the exception that 

the object support is different and the processing 
operations performed by the object/support separator 90 
are different, as will be described below. In addition, 
as in the second embodiment, texture data for the 

20 underside of subject object 210 is not generated in the 

third embodiment. 

In the third embodiment, the object support upon which 
the subject object 210 is placed for imaging comprises a 
25 cylinder, as in the second embodiment, but, in the third 
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embodiment, the whole of the cylinder is opaque rather 
than part being opaque and part being transparent as in 
the second embodiment. 



5 Consequently, as in the first embodiment, the 3D computer 

model generated by surface modeller 70 at step S6-10 
comprises a 3D computer model of the subject object 210 
together with the object support. 

10 At step S6-14 in the third embodiment, object/support 

separator 90 performs processing to test the cross- 
sectional area of the 3D computer model at different 
positions in a direction moving away from the position of 
the photographic mat 34 previously calculated by camera 

15 calculator 50 at step S6-6 (that is, in an upwards 

direction from the base of the object support to the 
subject object 210). More particularly, object/support 
separator 90 monitors the cross-sectional area of the 3D 
computer model in order to detect the position at which 

20 a sudden change in the cross-sectional area occurs (this 

sudden change indicating the boundary between the object 
support and the subject object 210 provided that the base 
of the subject object 210 is narrower or wider than the 
top surface of the object support). Having detected the 

25 position at which the cross-sectional area of the 3D 
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computer model changes, object/support separator 90 
performs processing to re-generate the 3D computer model 
as described in the first embodiment by setting the base 
plane 412 of the initial volume 400 on which the voxel 
carve processing is performed to be the detected position 
at which the cross-sectional area of the 3D computer 
model changes. 

In this way, a 3D computer model of the subject object 
210 alone is generated without user input. 

As an alternative to the processing described above, an 
object support having a substantially uniform colour 
which is substantially different to any colour at the 
bottom of the subject object 210 may be used and, instead 
of performing processing to detect a sudden change in the 
cross-sectional area . of the object support, 
object/support separator 90 may perform processing to 
detect the position of a sudden change in colour of the 
3D computer model when rendered with texture data 
generated by surface texturer 80. 

In the first three embodiments described above, the 3D 
computer model of the subject object 210 and object 
support is generated (either as a single 3D computer 



58 2714301 
model as in the first and third embodiments or as two 
separate 3D computer models as in the second embodiment) 
and the 3D computer model relating to the object support 
is subsequently removed. 

However, instead, surface modeller 70 may perform 
processing at step S6-10 to generate data defining a 3D 
computer model of the subject object alone, as will be 
clear from the fourth embodiment described below. 

Fourth Embodiment 

A fourth embodiment of the invention will now be 
described. The components of the fourth embodiment and 
the processing operations performed thereby are the same 
as those in the first embodiment, with the exception that 
the height of the object support is known and is stored 
in processing apparatus 2 (for example the height may be 
input by the user), and object/ support separator 90 (and 
consequently processing step S6-14 performed thereby) is 
not included and instead the processing performed by 
surface modeller 70 is modified. These differences will 
be described below. 



More particularly, because the height of the object 



support is known and stored in processing apparatus 2, at 
step S6-10, surface modeller 70 sets the base plane 412 
of the initial volume 400 on which to perform voxel carve 
processing to be at a vertical distance away from the 
position of the photographic mat 34 equal to the stored 
height of the object support. In this way, in the 
subsequent processing performed by surface modeller 70 at 
step S6-10, a 3D computer model of the subject object 210 
alone is generated. 

Modifications 

Many modifications can be made to the embodiments 
described above within the scope of the claims. 

For example, modifications can be made to the object 
support upon which the subject object 210 is placed for 
imaging. 

By way of a first example, the object support can be 
designed to assist the user in selecting an optimum 
position and viewing direction for camera 230 since, as 
described above with reference to Figures 4a-4e, the 
imaging position and direction of the camera 230 are 
important to ensure that the subject object 210 appears 
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surrounded by the background screen 228 in each input 
image. More particularly, referring to Figure 16, to 
assist the user in achieving the correct imaging position 
and direction, an object support 490 may be used having 
a marker 500 thereon so that the user can select the 
position and direction of the camera 230 such that the 
marker 500 appears at the boundary between the 
photographic mat 34 and the background screen 228 - the 
marker 500 being positioned at a sufficient distance away 
from the top of the object support such that when this 
alignment occurs, the subject object 210 will always be 
surrounded by the background screen 22 8- 

Ref erring to Figure 17, the object support and 
photographic mat 34 may be combined by providing an 
object support 510 on which the pattern of features from 
the photographic mat 3 4 is printed (or otherwise marked) 
at a position separated from the top surface of the 
object support on which the subject object 210 sits. 

In addition to providing features on the photographic mat 
34 for detection in the input images, as in the 
embodiments described above, features may also be 
provided on the object support, and processing may be 
performed by camera calculator 50 to detect the features 
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on the object support in each input image and to match 
the detected features between input images. The 
positions of the detected and matched features on the 
object support may then be used together with the 
detected positions of the features on the photographic 
mat 34 to calculate the positions and orientations of the 
input images. This is likely to increase the accuracy of 
the calculated positions and orientations (and hence lead 
to a more accurate 3D computer model of the subject 
object) because the features now being used to 
calculate the positions and orientations do not all lie 
in a single plane (which is the case in the embodiments 
described above because all of the features are on the 
photographic mat 34) and instead the features on the 
object support can be provided at different heights 
thereon. As an example of the way in which the detected 
features on the photographic mat 34 and the detected 
features on the object support may be used to calculate 
the positions and orientations of the input images, 
processing may be performed based on that described in 
EP-A-0898245 using the matches between features on the 
photographic mat 34 as "user-identified" matches and the 
matches between the features on the object support as 
"calculated" matches . 
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in addition, the features on the object support may be 
provided at known relative positions thereon. In this 
way, the positions and orientations of the input images 
may be calculated with further increased accuracy by 
using the known relative positions to constrain the 
solution that each calculated position and orientation 
can take in a conventional manner (that is, by requiring 
the calculated position and orientation to give imaging 
conditions which preserve the relative positions of the 
features on the object support). 

Yet further, the features on the object support may be 
provided at known relative positions and in addition the 
object support and photographic mat may be arranged to 
interconnect so that, when connected, the positions of 
the features on the object support are known relative to 
the positions of the features on the photographic mat. 
More particularly, referring to Figure 18, a photographic 
mat 520 may be used comprising a pattern of features on 
a slab of material having a hole 530 therein. An object 
support 540 having a plurality of features 560 each with 
a respective different colour and arranged at known 
relative positions interconnects with the photographic 
mat 520 by inserting the base thereof in the hole 530. 
The relative positions of the features on the 



photographic mat 520 and the features 560 on the object 
support 540 are fixed when the object support 540 is 
inserted in the hole 530 because a protrusion 550 is 
provided on the object support 540 which engages a notch 
in the hole 530. 

In the embodiments described above, the calibration 
object on which the object support and subject object 210 
are placed has the form of a two-dimensional photographic 
mat 34. However, instead, a three-dimensional 

calibration object may be used. For example, referring 
to Figure 19, a calibration object 570 having the form of 
a cube with a pattern of features on the vertical sides 
thereof may be used. 

The object support may be telescopic to allow the height 
thereof to be changed. 

The object support may take the form of a spike to be 
inserted into the subject object 210. 

In all of the embodiments described above, the object 
support has a flat top surface on which the subject 
object 210 sits. However, the subject object may be 
supported by separate pillars. For example, in the first 



embodiment, the horizontal top plate 222 of the object 
support may be removed and the subject object supported 
on the legs 224. In embodiments in which texture data is 
to be generated for the underside of the subject object, 
this arrangement would provide a reflection of at least 
part of the underside in the mirror 140 unaffected by 
distortion resulting from transmission of light through 
the surface on which the subject object is sitting (for 
example surface 222 in the first embodiment). 

In all of the embodiments described above, the features 
of the pattern on the photographic mat 34 are arranged 
around an empty central area in which the object support 
is placed. However, the features may be provided over a 
photographic mat 3 4 without a space for the object 
support, and the object support may be then placed over 
some of the features. This will not affect the 
processing operations to calculate the position and 
orientation of each input image provided that a minimum 
of six features are visible in each input image. 

In the embodiments described above, the object support 
with the subject object 210 thereon is positioned on the 
photographic mat 34 so that it is surrounded by the 
features in the pattern on the photographic mat. 
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However, the object support with the subject object 210 
thereon may be positioned on the photographic mat. 34 so 
that it is outside the pattern of features (for example 
in a corner of the photographic mat 34 outside the circle 
of features) - Similarly, the object support with the 
subject object 210 thereon need not be positioned on the 
photographic mat. For example, if camera 2 30 is fixed, 
the photographic mat 34 may be placed on a larger sheet 
of paper, the object support with the subject object 210 
thereon may also be placed on the larger sheet but off 
the photographic mat 34, and the larger sheet may be 
rotated and/or moved between the recording of images to 
provide different relative positions and orientations 
between on the one hand the camera 2 30 and on the other 
hand the photographic mat 34, object support and subject 
object 210. In this way, even though the subject object 
210 is not directly over the pattern of features on the 
photographic mat 34, the subject object 210 is still held 
above the pattern by the object support, and both the 
subject object 210 and the pattern will be visible in 
recorded images. Accordingly, the word "above" used 
herein should not be interpreted to mean that the subject 
object is directly over the photographic mat in a 
vertical direction . 
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in all of the embodiments described above, a 3D computer 
model of the subject object 210 alone is generated. 
However, for some applications, a 3D computer model of 
the subject object 210 together with the object support 
(or part thereof) may be acceptable. In such a case, it 
is unnecessary to perform processing to remove the 3D 
computer model of the object support. 

In the first embodiment described above, in the 
processing performed by object/support separator 90 at 
step S6-14, the plane 450 displayed to the user for 
movement to separate the subject object 210 from the 
object support 2 20 has a width which exceeds the width of 
any part of the 3D computer model of the subject object 
210. Similarly, before re-performing voxel carve 
processing, object/support separator 90 moves the base 
plane 412 of the initial volume 400 to remove the entire 
bottom portion of the volume 400 beneath the base plane 
412. This processing is satisfactory when, as shown in 
Figure 5, every part of the subject object 210 is above 
the top of the object support 220. However, a situation 
may arise in which part of the subject object 210 
overhangs the top of the object support 220 so that, 
using the processing described in the first embodiment, 
it would not be possible to separate the subject object 



210 and object support 220 using the plane 450. 
Accordingly, as an alternative, at step S6-14, 
object/support separator 90 may display a plane 450 
having the same shape and cross-sectional area as the 
object support 220 and, when amending the initial volume 
400 on which to perform voxel carve processing, instead 
of moving the base plane 412, a hole may be created in 
the volume 400 having a cross-sectional area and shape 
corresponding to that of the plane 450 displayed to the 
user and extending from the original base plane 412 to a 
position corresponding to the final position of the plane 
450 stored at step Sll-6. The initial volume 400 may 
also be amended in a similar way in the third and fourth 
embodiments to take account of situations in which the 
subject object 210 overhangs the top of the object 
support . 

As an alternative method of performing processing to 
remove the 3D computer model of the object support from 
the 3D computer model of the subject object and object 
support together, a 3D computer model of the object 
support may be pre-generated and pre-stored in processing 
apparatus 2 (for example by performing the processing 
described previously without the subject object 210 
present when the input images are recorded) and then 



processing may be performed by object/ support separator 
90 to match the pre-stored model of the object support to 
part of the 3D model of the subject and object support 
together, and to remove the detected matching part. 

In the embodiments described above, the printer 8 prints 
photographic mat 34 on a single sheet of paper. However, 
mat generator 3 0 may control printer 8 to print the 
photographic mat on separate sheets of paper, which can 
then be placed together to form the photographic mat 34. 

Rather than printing the photographic mat on a recording 
medium or displaying the photographic mat on display 
panel 10, the features of the photographic mat may be 
marked (for example painted) on a surface such as a 
floor. 

In the first embodiment described above, in order to 
generate texture data for the underside of the subject 
object 210, the sheet of paper on which the photographic 
mat is printed has a hole 130 cut therein and is attached 
to a mirror 140. However, instead, a mirror, or other 
suitably reflective material, may be provided on top of 
the sheet of paper at a position corresponding to that 
where the hole 130 would have been cut. Similarly, if 
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the photographic mat is displayed on display panel 10 , a 
mirror or other reflective material may be placed on the 
display panel. Alternatively, the pattern of features of 
the photographic mat may be printed, or otherwise marked 
on a mirror or other reflective surface. 

In the first embodiment described above, in the 
processing performed by object/support separator 90 at 
step S6-14, the 3D computer model generated by surface 
modeller 70 is amended by moving the base plane 412 of 
the initial volume 400 on which voxel carve processing is 
to be performed to a position corresponding to the 
position of plane 450 previously moved by the user. In 
addition, the same technique may be used to amend the 3D 
computer model to correct any errors therein. More 
particularly, referring to Figure 20a, processing 
apparatus 2 may display an image of the 3D computer model 
together with a plane 600 which can be moved by a user 
using a user input device 6 to any position and 
orientation. In this way, the user can move the plane 
600 to a position which separates any unwanted part of 
the computer model, such as part 610 in Figure 20a, which 
protrudes from the wanted part. As in the processing 
described above with reference to Figure 11, processing 
apparatus 2 stores the final position of the plane 600 
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moved by the user and, referring to Figure 2 0b, amends 
the initial volume 400 on which voxel carve processing is 
to be performed by setting a plane 620 at a position in 
the volume 40(f corresponding to the position of the plane 
600 defined by the user, and removing the portion 630 
from the initial volume 400 excluded by the plane 620. 
Processing apparatus 2 then re-performs the voxel carve 
processing and, as a result, the unwanted portion 610 of 
the 3D computer model is removed because this part lies 
in the volume 630 excluded from the voxel carve 
processing. 

In the embodiments described above, at step S6-4, data 
input by a user defining the intrinsic parameters of 
camera 230 is stored. However, instead, default values 
may be assumed for some, or all, of the intrinsic camera 
parameters, or processing may be performed to calculate 
the intrinsic parameter values in a conventional manner, 
for example as described in "Euclidean Reconstruction 
From Uncalibrated Views " by Hartley in Applications of 
Invariance in Computer Vision, Mundy, Zisserman and 
Forsyth eds, pages 237-256, Azores 1993. 

In the embodiments described above, image data from an 
input image relating to the subject object 210 and object 
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support is segmented from the image data relating to the 
photographic mat 34, surface 200 and background screen 
228 as described above with reference to Figure 7. 
However, other conventional segmentation methods may be 
used instead. For example, a segmentation method may be 
used in which a single RGB value representative of the 
colour of the photographic mat 34, surface 200 and 
background screen 22 8 is stored and each pixel in an 
input image is processed to determine whether the 
Euclidean distance in RGB space between the RGB 
background value and the RGB pixel value is less than a 
specified threshold . 

In the embodiments described above, the processing at 
steps S6-10 and S6-14 to generate data defining a 3D 
computer model is carried out using a voxel carving 
technique. However, other techniques may be used, such 
as a voxel colouring technique for example as described 
in University of Rochester Computer Sciences Technical 
Report Number 680 of January 1998 entitled "What Do N 
Photographs Tell Us About 3D Shape?" and University of 
Rochester Computer Sciences Technical Report Number 692 
of May 1998 entitled "A Theory of Shape by Space 
Carving", both by Kiriakos N. Kutulakos and Stephen M. 
Seitz . 
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In the embodiments described above, processing is 
performed by a computer using processing routines defined 
by programming instructions. However, some, or all, of 
the processing could be performed using hardware. 
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CLAIMS 

1. A method of recording images of a subject object 
from different positions and orientations and processing 

5 the recorded image data to generate a three-dimensional 

computer model of the subject object, comprising: 

supporting the subject object above a calibration 
object having a predetermined pattern of features using 
an object support; 
10 recording at different positions and orientations a 

plurality of images of the subject object supported above 
the calibration object; 

processing the recorded image data to calculate the 
position and orientation at which each of at least some 
15 of the images were recorded; and 

performing processing using the calculated positions 
and orientations to generate data defining a three- 
dimensional computer model of at least the subject 
object . 

20 

2. A method according to claim 1, wherein the images of 
the subject object supported above the calibration object 
are recorded with a background of a substantially uniform 
colour behind the subject object such that, in each 

25 recorded image, the outline of the subject object is 



surrounded by the background except where the outline 
touches the support. 

3. A method according to claim 2, wherein the 
background is provided by a background screen. 

4. A method according to claim 2 or claim 3, wherein at 
least the part of the object support adjacent the subject 
object is substantially the same colour as the 
background, and wherein the processing to generate the 
three-dimensional computer model comprises generating 
data defining a three-dimensional computer model of the 
subject object and a separated three-dimensional computer 
model of the object support. 

5. A method according to any of claims 1 to 3 , wherein 
at least the part of the object support adjacent the 
subject object is substantially transparent, and wherein 
the processing to generate the three-dimensional computer 
model comprises generating data defining a three- 
dimensional computer model of the subject object and a 

separated three-dimensional computer model of the object 

support . 



A method according to any preceding claim, wherein 
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the subject object is supported by at least one surface 
of an object support standing on the calibration object, 
and wherein each surface of the object support supporting 
the subject object does not protrude substantially from 
beneath the subject object. 

7. A method according to any preceding claim, wherein 
the object support has calibration features thereon, and 
wherein the processing to calculate the position and 
orientation at which each of at least some of the images 
were recorded includes detecting calibration features on 
the object support in image data and using the positions 
of the detected features to calculate the positions and 
orientations at which the images were recorded. 

8. A method according to claim 7, wherein data defining 
the relative positions of the calibration features on the 
object support is prestored and used to calculate the 
positions and orientations at which the images were 
recorded . 

9. A method according to claim 7 or claim 8, wherein 
the object support is arranged relative to the 
calibration object in a predetermined configuration, and 
wherein data defining the positions of the calibration 



features on the object support relative to the positions 
of the features on the calibration object is prestored 
and used to calculate the positions and orientations at 
which the images were recorded, 

10. A method according to any preceding claim, wherein 
the subject object is supported above a reflective 
surface, and wherein processing is carried out to 
generate texture data for the three-dimensional computer 
model of the subject object in dependence upon image data 
that corresponds to reflections in the reflective 
surface. 

11. A method according to any preceding claim wherein 
the calibration object is three-dimensional. 

12. A method according to any preceding claim, wherein 
the object support and the calibration object are formed 
as one, with the subject object being supported thereby 
above the predetermined pattern of features thereon. 

13. A method of processing image data defining a 
plurality of images recorded at different positions and 
orientations of a subject object supported by an object 
support above a calibration object having a predetermined 



pattern of features, comprising: 

processing the image data to calculate the positions 
and orientations at which at least some of the images 
were recorded, and 

performing processing using the calculated positions 
and orientations to generate data defining a three- 
dimensional computer model of the subject object but not 
the object support. 

14. A method according to claim 13, wherein the step of 
calculating the positions and orientations at which at 
least some of the images were recorded includes detecting 
matching features in the image data defining respective 
images corresponding to features on the object support. 

15. A method according to claim 13 or claim 14, wherein 
the step of generating the data defining the three- 
dimensional computer model comprises: 

processing images to segment image data relating to 
at least the subject object from background image data; 
and 

processing the segmented image data and the 
calculated positions and orientations to generate the 
data defining the three-dimensional computer model. 
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16 . A method of processing image data to generate a 
three-dimensional computer model , comprising : 

receiving image data defining at least in part a 
plurality of images of a subject object supported by an 
object support recorded at different relative positions 
and orientations ; 

receiving data defining the positions and 
orientations at which the images were recorded; and 

processing the received data to generate data 
defining a three-dimensional computer model of the 
subject object but not the object support. 

17. A method according to any of claims 13 to 16, 
wherein the step of generating data defining the three- 
dimensional computer model comprises performing 
processing using at least one known parameter of the 
object support to generate data defining the three- 
dimensional computer model of the subject object without 
generating data defining a three-dimensional computer 
model of the object support. 

18. A method according to claim 17, wherein the known 
parameter of the object support is the height of the 
object support . 
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19. A method according to claim 18, wherein the step of 
generating data defining the three-dimensional computer 
model of the subject object includes: 

defining a volume of voxels in a three-dimensional 
space in dependence upon the known height of the object 
support such that the object support, but not the subject 
object is excluded from the volume; and 

removing voxels from the volume in dependence upon 
the image data . 

20. A method according to any of claims 13 to 16, 
wherein the step of generating data defining the three- 
dimensional computer model comprises performing 
processing to generate at least one three-dimensional 
computer model of the subject object and object support 
and performing processing to remove the three-dimensional 
computer model of the object support, 

21. A method according to claim 20, wherein the 
processing to remove the three-dimensional computer model 
of the object support is carried out in dependence upon 
signals input by a user defining the three-dimensional 
computer model to be removed. 



22. A method according to claim 21, wherein the 
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processing to remove the three-dimensional computer model 
of the object support comprises: 

(a) generating image data for display to a user defining 
an image of the three-dimensional computer model of the 
subject object and object support together with a plane 
moveable by the user; 

(b) receiving signals input by the user defining a 
position of the plane; 

(c) repeating steps (a) and (b) ; and 

(d) removing the three-dimensional computer model which 
lies on a predetermined side of the plane. 

23. A method according to claim 22, wherein: 

the subject object is supported on the top-most 
surface of an object support standing on the calibration 
object; 

the plane is generated so as to have an orientation 
in a substantially horizontal plane; and 

processing is performed to allow the user to move 
the position of the plane but not to change the 
orientation of the plane. 

24. A method according to claim 22 or claim 23, wherein 
the plane is generated with the same shape and cross- 
sectional area as the object support. 
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25. A method according to any of claims 22 to 24 , 
wherein the step of removing the three-dimensional 
computer model which lies on the predetermined side of 
the plane comprises: 

defining a volume of voxels in a three-dimensional 
space such that a boundary of the volume is at a position 
corresponding to the position of the plane; and 

removing voxels from the volume in dependence upon 
the image data. 



26. A method according to claim 21, wherein processing 
is carried out to generate a three-dimensional computer 
model of the subject object and a separate three- 
dimensional computer model of the object support, and 

15 wherein the step of removing the three-dimensional 

computer model of the object support comprises generating 
image data for display to the user defining at least one 
image of the three-dimensional computer models, receiving 
signals input by the user defining one of the three- 

2 0 dimensional computer models and removing one of the 

three-dimensional computer models in dependence upon the 
signals input by the user. 



25 



27. A method according to claim 20, wherein the 
processing to remove the three-dimensional computer model 
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f the object support includes processing to identify the 
three-dimensional computer model to remove without 
signals input by a user. 

28. A method according to claim 27, wherein processing 
is carried out to generate a three-dimensional computer 
model of the subject object and a separate three- 
dimensional computer model of the object support, and 
wherein processing is carried out to remove the three- 
dimensional computer model having a position closest to 
the position corresponding to the position of the 
calibration object. 

29. A method according to claim 27, wherein processing 
is carried out to test at least one property of the 
three-dimensional computer model at different positions 
and to remove a part of the three-dimensional computer 
model in dependence upon a position at which the tested 
property changes . 

30. A method according to claim 29, wherein at least one 
of the cross-sectional area and colour of the three- 
dimensional computer model is tested. 



31. A method according to claim 27, wherein data 
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defining a reference three-dimensional computer model of 
the object support is prestored, the reference model is 
compared against the three-dimensional computer model of 
the subject object and the object support to identify the 
part thereof which corresponds to the object support, and 
the identified part is removed. 

32. A method according to any of claims 13 to 31, 
wherein the step of generating the data defining the 
three-dimensional computer model includes generating 
texture data using the image data. 

33. A method according to claim 32, wherein the step of 
generating texture data includes processing the image 
data to identify data corresponding to a reflection of 
the subject object in a reflective surface, and using the 
identified data to generate texture data for a surface of 
the three-dimensional computer model. 

34. A method according to claim 16, wherein the received 
image data comprises image data relating to the subject 
object and object support previously segmented from other 
image data in the recorded images. 



35. A method according to any of claims 13 to 34, 
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further comprising the step of generating a signal 
conveying data defining the three-dimensional computer 
model of the subject object. 

36. A method according to claim 35, further comprising 
the step of recording the signal either directly or 
indirectly . 

37. A method of processing images of an object to 
generate data defining a three-dimensional computer model 
of the object and to amend the three-dimensional computer 
model, comprising: 

(a) receiving image data defining at least in part a 
plurality of images of an object recorded at different 
positions and orientations ; 

(b) receiving data defining the positions and 
orientations at which the images were recorded; 

(c) processing the received data to generate data 
defining a three-dimensional computer model of the 
object; 

(d) generating data for display to a user defining an 
image of the three-dimensional computer model together 
with a plane moveable by the user; 

(e) receiving signals input by the user defining a 
position and orientation of the plane; 
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(f) repeating steps (d) and (e); and 

(g) removing the part of the three-dimensional computer 
model which lies on one side of the plane. 

38. A method according to claim 3 7 , wherein step (g) 
comprises processing image data received in step (a) and 
position and orientation data received in step (b) to 
generate data defining a three-dimensional computer model 
of the object in a three-dimensional space having a 
boundary defined by the plane. 

39. A system for recording images of a subject object 
from different positions and orientations and for 
processing the recorded image data to generate a three- 
dimensional computer model of the subject object, 
comprising: 

a calibration object having a predetermined pattern 
of features ; 

an object support for supporting the subject object 
higher than the calibration object; 

an imager for recording at different positions and 
orientations a plurality of images of the subject object 
supported higher than the calibration object; and 

an image data processing apparatus, comprising: 

means for processing the recorded image data to 



86 2714301 
calculate the position and orientation at which each of 
at least some of the images were recorded; and 

means for performing processing using the calculated 
positions and orientations to generate data defining a 
three-dimensional computer model of at least the subject 
object . 

40. A system according to claim 39, further comprising 
a screen having a substantially uniform colour for 
placing behind the subject object so that images of the 
subject object supported higher than the calibration 
object can be recorded with the screen behind the subject 
object such that, in each recorded image, the outline of 
the subject object is surrounded by the background except 
where the outline touches the support. 

41. A system according to claim 40, wherein: 

at least the part of the object support adjacent the 
subject object when the subject object is placed thereon 
is substantially the same colour as the screen; and 

the processing means for generating the three- 
dimensional computer model comprises means for generating 
data defining a three-dimensional computer model of the 
subject object and a separated three-dimensional computer 
model of the object support . 
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42. A system according to claim 39 or claim 40, wherein: 
at least the part of the object support adjacent the 

subject object when the subject object is placed thereon 
is substantially transparent; and 

the processing means for generating the three- 
dimensional computer model comprises means for generating 
data defining a three-dimensional computer model of the 
subject object and a separated three-dimensional computer 
model of the object support. 

43. A system according to any of claims 39 to 42, 
wherein the object support is arranged such that, when 
the subject object sits thereon, no surface supporting 
the subject object protrudes substantially from beneath 
the subject object. 

44. A system according to any of claims 39 to 43, 
wherein : 

the object support has calibration features thereon; 

and 

the processing means for calculating the position 
and orientation at which each of at least some of the 
images were recorded includes means for detecting 
calibration features on the object support in image data 
and using the positions of the detected features to 
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calculate the positions and orientations at which the 
images were recorded. 



45. A system according to claim 44, wherein the image 
5 data processing apparatus includes means for prestoring 

data defining the relative positions of the calibration 
features on the object support, and the processing means 
for calculating the positions and orientations at which 
each of at least some of the images were recorded is 
10 arranged to use the prestored data to calculate the 

positions and orientations at which the images were 
recorded . 

46. A system according to claim 44 or claim 45, wherein: 
15 the object support is arranged to connect to the 

calibration object in a predetermined configuration; 

the image data processing apparatus includes means 
for prestoring data defining the positions of the 
calibration features on the object support relative to 
20 the positions of the features on the calibration object 

when the object support is connected to the calibration 
object; and 

the processing means for calculating the positions 
and orientations at which each of at least some of the 
2 5 images were recorded is arranged to use the prestored 
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data to calculate the positions and orientations at which 
the images were recorded. 

47. A system according to any of claims 39 to 46, 
5 wherein: 

the calibration object has a reflective surface; and 
the image data processing apparatus includes means 
for generating texture data for the three-dimensional 
computer model of the subject object in dependence upon 
10 image data that corresponds to reflections in the 

reflective surface . 

48. A system according to any of claims 39 to 47, 
wherein the calibration object is three-dimensional. 

15 

49. A system according to any of claims 39 to 48, 
wherein the object support and the calibration object are 
formed as one with a surface for supporting the subject 
object such that, when the subject object sits thereon, 

20 the subject object is supported higher than, and 

separated from, the predetermined pattern of features. 

50. Apparatus for processing image data defining a 
plurality of images recorded at different positions and 

25 orientations of a subject object supported by an object 



support higher than a calibration object having a 
predetermined pattern of features, comprising: 

means for processing the image data to calculate the 
positions and orientations at which at least some of the 
images were recorded, and 

means for performing processing using the calculated 
positions and orientations to generate data defining a 
three-dimensional computer model of the subject object 
but not the object support. 

51. Apparatus according to claim 50, wherein the means 
for calculating the positions and orientations at which 
at least some of the images were recorded includes means 
for detecting matching features in the image data 
defining respective images corresponding to features on 
the object support . 

52. Apparatus according to claim 50 or claim 51, wherein 
the means for generating the data defining the three- 
dimensional computer model comprises: 

means for processing images to segment image data 
relating to at least the subject object from background 
image data; and 

means for processing the segmented image data and 
the calculated positions and orientations to generate the 



data defining the three-dimensional computer model. 

53. Apparatus for processing image data to generate a 
three-dimensional computer model, comprising: 

means for receiving image data defining at least in 
part a plurality of images of a subject object supported 
by an object support recorded at different relative 
positions and orientations; 

means for receiving data defining the positions and 
orientations at which the images were recorded; and 

means for processing the received data to generate 
data defining a three-dimensional computer model of the 
subject object but not the object support. 

54. Apparatus according to any of claims 50 to 53, 
wherein the means for generating data defining the three- 
dimensional computer model comprises means for performing 
processing using at least one known parameter of the 
object support to generate data defining the three- 
dimensional computer model of the subject object without 
generating data defining a three-dimensional computer 
model of the object support. 

55. Apparatus according to claim 54, wherein the known 
parameter of the object support is the height of the 
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object support. 

56. Apparatus according to claim 55, wherein the means 
for generating data defining the three-dimensional 
computer model of the subject object includes: 

means for defining a volume of voxels in a three- 
dimensional space with the base plane of the volume set 
to be at a height higher than the calibration object 
corresponding to the known height of the object support; 
and 

means for removing voxels from the volume in 
dependence upon the image data. 

57. Apparatus according to any of claims 50 to 53, 
wherein the means for generating data defining the three- 
dimensional computer model comprises means for performing 
processing to generate at least one three-dimensional 
computer model of the subject object and object support 
and means for performing processing to remove the three- 
dimensional computer model of the object support. 



58. Apparatus according to claim 57, wherein the means 
for performing processing to remove the three-dimensional 
computer model of the object support comprises means for 
removing the three-dimensional computer model in 



dependence upon signals input by a user defining the 
three-dimensional computer model to be removed. 

59. Apparatus according to claim 58, wherein the means 
for performing processing to remove the three-dimensional 
computer model of the object support comprises: 

means for generating image data for display to a 
user defining an image of the three-dimensional computer 
model of the subject object and object support together 
with a plane moveable by the user; 

means for receiving signals input by the user 
defining a position of the plane; and 

means for removing the three-dimensional computer 
model which lies on a predetermined side of the plane. 

60. Apparatus according to claim 59, wherein the subject 
object is supported on the top-most surface of an object 
support standing on the calibration object, and the means 
for performing processing to remove the three-dimensional 
computer model of the object support is arranged to 
operate so that: 

the plane is generated so as to have an orientation 
in a substantially horizontal plane; and 

processing is performed to allow the user to move 
the position of the plane but not to change the 
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orientation of the plane, 

61. Apparatus according to claim 59 or claim 60, wherein 
the means for performing processing to remove the three- 
dimensional computer model of the object support is 
arranged to operate so that the plane is generated with 
the same shape and cross-sectional area as the object 
support . 

62. Apparatus according to any of claims 59 to 61, 
wherein the means for removing the three-dimensional 
computer model which lies on the predetermined side of 
the plane comprises: 

means for defining a volume of voxels in a three- 
dimensional space such that a boundary of the volume is 
at a position corresponding to the position of the plane; 
and 

means for removing voxels from the volume in 
dependence upon the image data. 

63. Apparatus according to claim 58, wherein: 

the apparatus is arranged to carry out processing to 
generate a three-dimensional computer model of the 
subject object and a separate three-dimensional computer 
model of the object support; and 
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the means for removing the three-dimensional 
computer model of the object support comprises: 

means for generating image data for display to the 
user defining at least one image of the three-dimensional 
5 computer models; 

means for receiving signals input by the user 
defining one of the three-dimensional computer models; 
and 

means for removing one of the three-dimensional 
10 computer models in dependence upon the signals input by 

the user. 



64. Apparatus according to claim 57, wherein the means 
for performing processing to remove the three-dimensional 
15 computer model of the object support is arranged to 

operate to identify the three-dimensional computer model 
to remove without signals input by a user. 



65. Apparatus according to claim 64 , wherein the 
20 apparatus is arranged to carry out processing to generate 

a three-dimensional computer model of the subject object 
and a separate three-dimensional computer model of the 
object support, and the means for performing processing 
to remove the three-dimensional computer model of the 
25 object support is arranged to carry out processing to 



remove the three-dimensional computer model having a 
position closest to the position corresponding to the 
position of the calibration object. 

66. Apparatus according to claim 64, wherein the means 
for performing processing to remove the three-dimensional 
computer model of the object support is arranged to carry 
out processing to test at least one property of the 
three-dimensional computer model at different positions 
and to remove a part of the three-dimensional computer 
model in dependence upon a position at which the tested 
property changes . 

67. Apparatus according to claim 66, wherein the means 
for performing processing to remove the three-dimensional 
computer model of the object support is arranged to test 
at least one of the cross- sectional area and colour of 
the three-dimensional computer model to determine the 
part of the three-dimensional computer model to remove. 

68. Apparatus according to claim 64, wherein the means 
for performing processing to remove the three-dimensional 
computer model of the object support comprises: 

means for prestoring data defining a reference 
three-dimensional computer model of the object support; 
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means for comparing the reference model against the 
three-dimensional computer model of the subject object 
and the object support to identify the part thereof which 
corresponds to the object support; and 

means for removing the identified part. 

69. Apparatus according to any of claims 50 to 68, 
wherein the means for generating the data defining the 
three-dimensional computer model includes means for 
generating texture data using the image data. 

70. Apparatus according to claim 69, wherein the means 
for generating texture data includes means for processing 
the image data to identify data corresponding to a 
reflection of the subject object in a reflective surface, 
and means for using the identified data to generate 
texture data for a surface of the three-dimensional 
computer model - 

71. Apparatus according to claim 53, wherein the 
received image data comprises image data relating to the 
subject object and object support previously segmented 
from other image data in the recorded images. 



72. Apparatus for processing images of an object to 
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generate data defining a three-dimensional computer model 
of the object and for amending the three-dimensional 
computer model, comprising: 

means for receiving image data defining at least in 
part a plurality of images of an object recorded at 
different positions and orientations; 

means for receiving data defining the positions and 
orientations at which the images were recorded; 

means for processing the received data to generate 
data defining a three-dimensional computer model of the 
object; 

means for generating data for display to a user 
defining an image of the three-dimensional computer model 
together with a plane moveable by the user; 

means for receiving signals input by the user 
defining a position and orientation of the plane; and 

means for removing the part of the three-dimensional 
computer model which lies on one side of the plane. 

73. Apparatus according to claim 72, wherein the means 
for removing the part of the three-dimensional computer 
model comprises means for processing received image data 
and received position and orientation data to generate 
data defining a three-dimensional computer model of the 
object in a three-dimensional space having a boundary 
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defined by the plane. 

74. A storage device storing instructions for causing a 
programmable processing apparatus to become operable to 
perform a method as set out in at least one of claims 13 
to 38. 

75. A signal conveying instructions for causing a 
programmable processing apparatus to become operable to 
perform a method as set out in at least one of claims 13 
to 38. 
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In an image processing system, a subject object 210 is 
placed on top of an object support 220 on a calibration 
object 3 4 having a known pattern of features thereon. 
Images recorded at different positions and orientations 
are processed to generate a three-dimensional computer 
model of the subject object alone or the subject object 
together with the calibration object . By imaging the 
subject object 210 on an object support instead of 
placing it directly on the calibration object, the user 
is provided with flexibility in the selection of the 
pattern and colour of the calibration and the selection 
of the imaging positions and orientations. By providing 
an object support 220 having a top surface on which the 
subject object 210 sits which does not protrude from 
beneath the base of the subject object 210, the user is 
provided with flexibility in the selection of lighting 
conditions. By imaging the subject object 210 with a 
background screen 228 behind it, the user is provided 
with flexibility in the selection of the surface on which 
the calibration object 34 is placed for imaging. 
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Fig.4d. 
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Fig. 6 
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DISPLAY IMAGE OF THE 3D 
MODEL OF THE SUBJECT 
OBJECT AND SUPPORT 
TOGETHER WITH A MOVEABLE 
PLANE 
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Fig. 14. 
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